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Abstract 

Application or replication research, already rare, is diminishing in both 
quantity and quality, for a variety of reasons ("How science goes wrong," 2013; 
"For my next trick," 2016). In this study of replications and applications, 351 
papers that included a reference to any one of ten of the author's papers 
published between 2001 and 2007 (the "child" papers) were examined. A total of 
seventeen instances of application! replication by other researchers (the relative 
rarity already a finding) of some element, processes, procedures, instruments, or 
findings, of one of the author's "parent" papers were found, about 5% of the total 
351 original parent papers. No self-replications by the author were studied. 

The findings showed that, of the small number of replications, three (less 
than 18% of the total) were exact replications, five (about 29%) were partial, and 
nine (about 53%) were conceptual only ("investigating the same construct but 
with different methods and measures"; Jones, Derby, & Schmidlin, 2010). All of 
the replications reported were based on positive or neutral views of the parent 
paper, none were based on a negative view, potentially problematic, it was 
concluded, if one of the goals of replication is to identify weaknesses or mistakes 
in previous work. Another finding was that, after publishing in this area, a 
minority of authors published another paper on a related topic in the same area 
at least once in the future. The paper concluded with a call for greater 
acceptance and valuing of application! replication research in general, and of 
disagreements (and therefore corrections of concepts) among practitioners, by 
researchers, publishers, editors, reviewers, and authors. 
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Introduction 

Published research may be used in various ways by others - or may be 
ignored altogether. In a previous paper (Fahy, 2013), the author examined how 
ten of his papers (the parent papers) were, after publication, replicated or applied 
by others. In that paper, it was previously reported that: 

1) usage was general, but appeared to be very time-sensitive (all ten 
of the parent papers had been referenced at least twice in the 
published research of others; maximum usage of one paper was 72, 
and the mean number of citations was 33.7, but the older papers 
were referenced considerably more often); 

2) single references (citing just the author's paper) were far more 
common than multiple references (citing the paper is a list); 


3) applications/replications were the least common type of usage, 
followed by contrasts/elaborations, and quotations; 


4) unlike publications in the "hard" (laboratory) sciences, whether a 
parent paper was solo- or co-authored did not appear to affect usage; 


5) publication of a parent in a non-prestige journal, as defined in the 
literature, was actually associated with more usage than 
publication in a prestige journal; and, 

6) a large majority of uses of the parent papers (over 80%) were in 
heavily scrutinized or closely supervised sources ( journal articles, or 
theses and dissertations). 
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The present paper continues the previous work, examining the same 
corpus of publications further, focusing specifically on applications/replications of 
elements of the parent papers. 

Background 

Previous work on the uses and value of published research in the social 
sciences, including open and distance learning (ODL), is not extensive, but the 
existing literature suggests that ODL publications are in a poor state: the research 
is often poorly designed, conducted, or described (Keegan, 1985; Moore, 1985; 
Holdaway, 1986; Cannell, 1999; Saba, 2000; Gibson, 2003; Zawicki-Richter, 

Backer, & Vogt, 2009); academic papers as a class are often ignored by other 
scholars after publication (Lehrer, 2010); the conclusions can be rejected or 
proven wrong by subsequent research ("Publish and be wrong," 2008); papers 
may be retracted for a variety of reasons by their authors or publishers 
(Groopman, 2010); and published work may even be proven fraudulent ("Liar! 
Liar!", 2009). In the popular press, readers are warned that if they do not 
encounter further confirmation of published reports they might suspect the 
original findings "have fallen by the wayside" ("Journalistic deficit disorder," 
2012; Kaku, 2014; "Let just try that again," 2016). It is also not unusual for effects 
initially reported to decline when examined later, the result of hindsight and the 
application of more powerful research tools (Ioannidis, 2005; Coyne, 2010). Even 
research widely considered valid and robust is typically only occasionally cited 
(Abbott, 2007), often because it cannot be replicated by others (Jones et al., 2010; 
"University challenge," 2011). 


Retractions increased tenfold in the previous decade, and America's 
National Institutes of Health (NSH) reckon despairingly that researchers would 
find it hard to reproduce at least three-quarters of all published biomedical 
findings ("Trouble at the lab," 2013). Replication research is also viewed 
suspiciously by young researchers, who believe that "verification . . . does little to 
advance a researcher's career" ("How science goes wrong," 2013, p. 13). 

In its defense, the ideal of replication research is to refine and correct 
research concepts and perceptions through further examination, inquiry, and 
consideration (Jones et al., 2010; "A far from dismal outcome," 2016). It has long 
been recognized that replications and applications are an important way of 
assuring that good quality research is recognized, circulated, updated, 
corroborated, and applied (Jones & Charnley, 1978; Wilson, 2001, p. 38). 
Moonesinghe, Khoury, and Janssens (2007) observe, "... replication of research 
findings enhances the positive predictive value of research. ..." (p. 220). Adams 
et al. (2005, p. 56) acknowledge the power of further research/replications, and, 
given how common lack of corroboration in research is (p. 590), approvingly 
comment: "Replication is the test for objectivity and is at the very heart of any 
scientific discipline." They observe that empirical findings should not be widely 
accepted until they are retested and confirmed by multiple occurrences (p. 58). 
Adams and colleagues comment, "We have greater confidence in the replication 
effort when the same 'effects' are observed across samples by different 
observers" (p. 64). (A similar point is made by Carlson and Ji [2011, p. 711].) 

The role of replications in providing contrasting views is also recognized, 
often by focusing attention on contentious findings or issues. Derby et al. (2008, 
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p. 672), for example, report that papers with more "children" were more likely to 
be replicated. 

There are a variety of reasons why adaptations/replications are so rare in 
the social sciences (Gopnik, 2015). Adams et al. (2005) blame these causes: 

1. Corroboration of previous research is not a priority among agencies and 
individuals that fund research (Jones et al., 2010). National funding 
agencies routinely announce that they fund only "fundamental new 
knowledge," "cutting-edge" research, and "development of innovative 
methods and models," in proposals "that offer unique approaches." 
Generally, agencies (and publications - see #2, below) appear to regard 
replication research - which can be seen as a form of sober second thought 
- as derivative and therefore not worthy of funding. 

2. Reviewer and editor bias about replications in many social science 
journals is common, with the result that application research is rarely 
accepted for publication (Weisman, 2011; Jones et al., 2010, p. 587). 

3. As a consequence of the above, social scientific research is dominated by, 
and researchers are used to seeing printed, poor research: "stories, 
anecdotes, cross-sectional, and 'one-shot' studies” (Adams et al., 2005, p. 
62). 

Well thought-out remedies that might provide for more replications, and 
rationales for the publication of contrasting views in research, do exist in the 
literature, however. Bell, DiStefano, and Morgan (2010) argue that "transparency 
and replication are essential features of scientific inquiry"; they offer no fewer 
than ten suggestions on replication and revision, intended to produce more 
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accurate social science research over time. Conroy, Dunlap, Clarke, and Alter 
(2005) first observe in relation to social science research that "without replication 
across participants and researchers, the findings can be limited," then suggest 
that, to assure accuracy, before findings are accepted, "at least five single subject 
design studies [should be] conducted, across 20 participants with researchers 
across the country" (p. 164). Derby et al. (2008) suggest that application 
"demonstrates that a given finding can or cannot be repeated," which allows 
revision, correction, and the gradual accumulation of sound research knowledge. 
Freese (2007) agrees, stating that, because the credibility of quantitative social 
science rests on the belief that results reported by one researcher will be verified 
by others, concerns about replicability and contrasting views have increased (p. 
153). (Freese goes on to suggest the interesting notion that graduate students 
consider publishing such studies to increase useful publications at the start of 
careers, and to increase awareness and scrutiny of a range of published research 
findings.) 

Despite its importance, however, Helmig, Spraul, and Tremp (2012) note 
that replication research is still "largely missing" (p. 361) from the social sciences, 
and the reporting and publication of contrasting or contradictory findings is rare. 
(Jones et al., 2010 disagree in relation to their own field, concluding that 
"replications seem to be common in the human factors literature" [p. 587], They 
voice some criticisms of the research that has been reported, however.) Finally, 
Henson, Hull, and Williams (2010) also deplore "one-shot" studies as a way of 
establishing or corroborating any theory (p. 122). 

Researchers have recently become more critical of views or practices 
disparaging replication research, advocating the potential usefulness of this kind 
of research, especially for an audience of academic novices. A previously 
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mentioned article ("How science goes wrong," 2013) notes that it is an accepted 
rule of thumb among biotechnology venture capitalists that a huge plurality (up 
to half of the research published in academic journals) cannot be replicated (p. 

13). Leithwood and Sun (2012) approvingly refer to those who argue that 
dissertations and other unpublished works are good sources of the 95% of 
research that produces noteworthy, potentially interesting and important, but 
non-statistically-significant results. Abbott (2007) identifies a variety of problems 
that he believes might slow the progression of application/replication research (pp. 
215 - 216), causing this type of research to spread (if at all) only fitfully through 
the discipline (p. 210). Weiner, Halle, Lund, and Shoou-Yih (2011) note that the 
"publication outlet" is significant to the reception accorded qualitative research, 
making the views of journal editors and reviewers even more significant to 
researchers (especially the younger and newer ones), if they are negative toward 
replication research. 

To summarize: while applications/replications do occur in the social 
sciences, they are widely thought not to occur often enough; at the same time, 
researchers regard this type of work as potentially improving the validity, 
predictability, and consistency of concepts, methods, tools, and findings, and 
therefore of the confidence others have in published results; without revisiting 
and revising beliefs, some important research findings of a discipline may come 
to be based upon anecdotal or "one-shot" observations; replication research, 
though often resisted by editors and reviewers, is regarded as fundamental to the 
growth of the scientific elements of disciplines. 
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The study 

Google Scholar (http ://scholar . goo gle . ca/) often provides direct links to 
published research (the exception is books and book chapters, which at the time 
of this study were usually not available in full-text format through Google 
Scholar ), as well as providing specifics about usage by other researchers. The 
author was able to retrieve full-text copies of over 98% of the publications (parent 
and child) used in this study, through the various Athabasca University library 
subscription services. When full-text versions of the child papers were not 
available, they were removed from the analysis, since this situation was rare. 

The procedure used was as follows. First, the ten "parent" papers were 
identified, then Google Scholar was used to identify the 351 "child" papers that 
referred to one of these, originally published between 2000 and 2007. Of these, 
this study focused on the 17 child papers that contained at least one instance of 
application or replication (these 17 papers were about 5% of the 351 total child 
papers). 

The following questions were asked in relation to the total number of 
child papers: 

1. Flow often were applications/replications of parent paper materials 
observed in the child papers? 

2. What were the proportions in the applications/replications of 
complementary, parallel, null set, or antagonistic assessments, as 
determined by the researcher? 

3. When an application or replication of some element of the original 
parent paper was conducted, was the application/replication an exact, 
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partial, or conceptual application/replication of the original research (using 
definitions from the literature) ? 

4. How often did conducting and reporting research in this area lead a 
researcher to publish further in the same area? 

For manageability, the original literature searches were limited to the 
research databases EDITLib, ERIC, SAGE, Education Research Complete, and 
ProQuest Education Journals; consultation with Athabasca University library staff 
led to the conclusion that these five databases constituted the main sources for 
papers on open and distance learning (ODL). 

For a study to be considered a replication, it had to meet the requirements 
of the definition provided by Jones et al. (2010, p. 586): "When a study is 
methodologically or conceptually similar to an earlier one, then the later study is 
considered a replication.” More is said about the various possible types of 
replications below. 

Findings 

1: Frequency of applications/replications 

The first question was the frequency of the various types of 
applications/replications in the child papers, related to the definitions used by Jones 
et al. (2010). (Similar definitions and language are used by Adams et al. [2005]). 

Figure 1: Jones et al.'s (2010) definitions of types of application/replications 

• Exact applications/replications: "strive to find the same results with the same 
instruments" 

• Partial applications/ replications: "investigate the same construct" and are 
"an extension of previous research, . . . retaining most of the original 
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measures" 

• Conceptual applications/replications: "investigate the same construct, with 
different methods and measures" 

The frequency of each of the above is shown in Figure 2: 


Figure 2: Frequencies of types of replications in the present study 


Type of replication/ application 

Occurrences of applications or replications 


# 

O/ 

/o 

Exact 

3 

18 

Partial 

5 

29 

Conceptual 

9 

53 

TOTAL 

17 



Figure 2 shows that applications /replications were rare - less than 5% of the 
child papers. Figure 2 also shows that conceptual replications/applications were the 
most common among the child papers (more than half), and, at about a third, 
partial replications/applications were next most common. Exact 
replications/ applications were the least common among the child papers. One 
interpretation is that the researchers in this study usually used the parent paper 
as a developmental springboard for their own thoughts and work, but only 
rarely engaged in exact replications/applications in the form of duplications 
where the original thinking was borrowed and applied exactly. A positive 
conclusion was that the work done by most of the researchers included in this 
survey was more creative and original than imitative. 

On the other hand, while creativity might be laudable, and evidence of a 
lively curiosity, the rarity of exact applications/replications meant, for purposes of 
this study of replications, that original research findings were not usually 
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subjected to careful reexamination and reevaluation through the process of exact 
replication of approaches and instruments. For whatever reason, this study 
found that most child papers were not exact replications/applications of parent 
papers, and that the parent papers and their findings thus still remained to be 
duplicated precisely. 

2: Positive , neutral and negative endorsements of parent papers 

Do appliers of previous research follow the path that was 
reported/deplored in the literature, endorsing only studies with which they 
agree? Do researchers correct studies they feel are in error? An examination of 
the papers by the researcher was done to determine the answer; Figure 3 shows 
the results. 


Figure 3: Reaction to parent papers by users (Bloome, Carter, Christian, Madrid, 
Otto,Shuart-Faris, and Smith, 2008) 


Reactions 

Positive 

Neutral 

Negative 

9 (53%) 

8 (47%) 

0 


The above indicates strongly that the most common type of endorsement 
of the parent papers by researchers was positive, consistent with the literature 
finding that, often because of the preferences of editors and reviewers, the vast 
majority of published literature contains only positive findings (Ioannidis, 2005; 
Lehrer, 2010). Neutral views in this study consisted of simple recounting of what 
was done or found in the parent research, without a positive or negative 
viewpoint. As can be seen from Figure 2, overtly negative judgments of parent 
elements (processes, procedures, instruments, or findings) did not occur at all. 
Lack of adoption may be seen as reflecting a negative view, just as adoption may 
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be seen as a positive endorsement. These findings should be considered 
preliminary until, in the spirit of replication, additional study corroborates or 
further explains them. 

Besides other positive observations, it is noteworthy here that none of the 
user applications/replications resulted in the rejection of a finding or a process from 
a parent paper, and that none of the researchers reported a negative view about a 
parent paper. While this might be an instance of journal or reviewer bias in 
favour of positive results, as noted earlier, it may also be an indication that the 
parent papers were completely useful to users, or that the findings of the parent 
work completely passed scrutiny by subsequent researchers. (This question 
might profitably be further researched.) 

Research is usually based on a positive (or at least a neutral) view of the 
research being cited ("Trouble at the lab," 2013); researchers rarely cite material 
with which they disagree, or in which they see problems. However, Bloome et 
al. (2008, p. 129) identify four stances research can take toward its predecessors, 
beyond the perhaps simplistic categorization used here: complementary, parallel, 
null set, and antagonistic. In complementary research, according to these authors, 
there is "compatibility with regard to the underlying theoretical assumptions 
about language, people, and knowledge"; parallel research uses "similar 
dimensions or aspects but not necessarily complementarity of underlying 
theoretical assumptions"; null set research perspectives "appear on the surface to 
have some relationships to each other, but upon deeper inspection are not at all 
about the same thing; upon deeper reading they are so different that they are not 
at all studying the same thing or using similar approaches"; and antagonistic 
research contains "mutually exclusive theoretical constructs across research 
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perspectives" (p. 129). This typology should be applied with a larger sample in 
future research to determine the suitability of the categories, and to explore the 
question further. 

Two findings in this area are of particular note, because they reflect on 
what the literature had predicted: 1) none of the user applications/replications 
resulted in questioning - much less rejection - of a finding or a process of a 
parent paper; and, 2) none of the follow-up research focused precisely on an 
aspect of the original parent research that was deemed deficient in some way. As 
noted earlier, while this might be another instance of the bias of journal editors 
and reviewers it may also be an indication that the parent papers were in fact 
consistently and uniformly useful, were regarded positively by users, and that 
the findings of the original parent work met its objectives and subsequently 
passed careful scrutiny. Here, as noted earlier, the conclusion was reached that 
researchers who wrote the child papers consistently and uniformly did so only if 
they regarded elements of the original parent paper positively (they endorsed the 
results) or, at worst, were neutral about or in parallel with them (they mentioned 
some of the parent papers' results, but did not explicitly refute or endorse them). 

Citations, however, imply an endorsement, even if a neutral or parallel 
one, in the terms used here (and of the concept of complementarity found in 
Bloome et al., 2008), and the endorsement is definite if the results cited are 
positive. For example: 

"As Fahy et al. (2001, p. 2) state: 'Practitioners and researchers must be able to 
describe online interaction more than impressionistically and measure them 
more effectively than anecdotally.'" [Cook & Raison, 2003, p. 361] 
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"This study adopted TAT (Fahy, 2001) as the analysis technique to analyze 
students' patterns of interaction and online supportive behaviours." [Lin, Yueh, 
Liu, Shoji, Kakusho, & Minoh, 2007, p. 2] 

Neutral expressions made no judgment about what they reported: 

"In addition, researchers have already reported moderate inter-rater reliabilities 
for the TAT with Cohen kappa's ranging from 0.45 to 0.65 (Fahy, Crawford, & 
Ally, 2001)." [de Leng, 2009, p. 98] 

"Gender appears to influence the way in which individuals interact and 
communicate in virtual environments (Fahy, 2001; Yates, 1996; Herring, 

1996)." [Lawlor, 2004, p. 2] 

As noted above, reasons for the lack in the child papers of negative 
interpretations of anything in the parent papers remains to be studied and 
explained further. 

3: Further research in the area of replication/application 

In relation to the third question, five online Athabasca University library 
databases listed above were consulted to determine whether conducting and 
reporting research in any area led to further research in the same area by the author, as 
indicated by further publication on the same or a related topic. To be recognized as 
further publishing in the same area, a subsequent paper had to meet completely 
the following three standards: 

1. The title or abstract had to indicate that a subsequent child paper was 
in the same area or in an area closely related to the 
application/replication research. 
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2. The subsequent child paper had to cite at least one of the parent 
papers. 

3. The original author had to be the primary author of both the original 
application/replication paper, and of further published papers in the 
same or similar area as the application I replication. 

A search of the five databases showed that 13 authors were associated 
with publishing a total of 17 additional papers, as solo or lead author; four of 
these example authors subsequently published in the same area more than once. 
Figure 4 shows the most prolific authors. 


Figure 4: Further research by the study's four most published authors 


Solo/primary authors 

(total 

application/replication 
authors: 13) 

References to parent 
papers 

(total references: 959) 

Subsequent papers 
authored/co-authored in 

the same or similar areas 

(total papers: 17) 


References 

%of 

total 

Papers 

% of total 

Murphy, E. 

28 

2.9% 

6 

35% 

Oriogun, P. 

70 

7.3 

5 

29 

Schellens, T. 

53 

5.5 

2 

12 

Valcke, M. 

53 

5.5 

2 

12 

TOTAL 

204 

21.3% 

15 

88% 


Based on the above it was concluded that, while only a minority 
continued to work in the field of application/replication research, those who did 
contributed extensively (the mean number of subsequent publications was 3.75 
per researcher, the median was 3.5 papers). 
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Regarding applications/replications 

As noted earlier, there is consensus among science researchers that 
applications and replications are at the heart of the field's research, providing 
confirmation of objectivity and the lack of obvious bias or error in the reported 
results, therefore predicting the likely enduring validity and applicability of 
published papers (Neuliep & Crandall, 1993; Adams et al., 2005; Conroy et al., 
2005; Brandon & Singh, 2009; Chrisman, Kellermanns, Chan, & Liano, 2009; Jones 
et al., 2010; Carlson & Ji, 2011; "For my next trick," 2016). Replication! application 
studies may not be identified as such in the published literature (Jones et al., 

2010, p. 588, 594), a tendency confirmed here. As noted earlier, self -replications of 
the author's own previous research, thought to be more common, are also often 
not identified, to the frustration and detriment of some (Adams et al., 2005, p. 55; 
Jones et al., 2010, p. 186; McDonnell, Stanton, & Burgess, 2011). 

On exact replications! applications. Though these were rare (at 17.6%) they 
were not particularly unusual in this study (the number was 3 of 17, of a very 
small sample). Clearly, the occurrence of replications/applications in other 
situations deserves further study. Partial and conceptual replications/applications 
are "more prevalent" in the literature (Jones et al., 2010), and were more common 
here (82%). The relative rarity of exact or literal applications! replications may be 
due to the lack of identical research circumstances that would permit exact 
replications, or to the desire of researchers to put their individual stamp on uses 
of research tools and methods. Both of these are potentially valid reasons for 
conducting partial or conceptual research; confirmation of their existence, and 
the reasons for exact applications and replications in ODL, deserve further 
examination in future studies. 
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The finding that a great majority of applications and replications were based 
on positive, or at worst neutral, views of the parent paper's processes, 
procedures, instruments, or findings, may confirm one of the literature's 
warnings, that publication often assumes positive (or at least not negative) 
results from applications and replications (Conroy et al., 2005; Ioannidis, 2005; 
Lehrer, 2010). Among the reasons for this phenomenon, as noted previously, 
may be the preferences - or requirements - of editors, reviewers, and funding 
agents (Patton, 1975; Abbott, 2007; Derby et al., 2008; Jones et al., 2010; Weiner et 
al., 2011; Weisman, 2011), as well as the predominance of genuinely positive 
outcomes noted in replication/application research. This finding also deserves 
further examination in further work. 

It also appeared that, while the types of replications/applications ranged 
from conceptual to full/exact, ODL researchers who replicated previous work did 
not do so slavishly here. As noted earlier, the tendency to adapt previous 
elements when they replicated them, and to be creative in the role of replicator, 
can be seen as a sign that ODL researchers used previous work more as a starting 
point (described earlier as a "springboard") than as a limitation to their research. 
In this study, researchers who engaged in replications or applications only 
occasionally borrowed the processes, procedures, and instruments of the parent 
researcher in toto; more often, they made adjustments of various kinds ( partial 
implementation), or merely used the previous work for further development 
(i conceptual application/replication). This last case suggests that researchers were 
quite sensitive to differences in what the parent paper had examined, and the 
focus of their own work, a positive development - and, ironically, contrary to the 
purposes and concerns of replication research. 
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A final word needs to be said to researchers, as a result of these findings: 
Researchers should consider using the work of others more often in their own 
research, and when they do they should look for corroboration that previous 
findings are valid, or remain valid, in any new or untried circumstances, in the 
spirit of a genuine replication. Practitioners apparently recognized that, as 
circumstances change, tools and media should, too. The assumption that 
changed realities do not invalidate previous knowledge needs to be confirmed, 
or at least tested, as one of the roles of research and researchers. 

Conclusions and implications 

In this study, applications and replications occurred only about 5% of the 
time. (Freese's [2007, p. 168] definition of replication was applied: "... using the 
same data and procedures with the goal of verifying results"; "attempting to see 
if the same findings are observed in a fully repeated study ...."). That such a 
small percentage of the elements of previously published papers was subjected 
to further scrutiny through application/replication may be regarded as perplexing, 
since this means that more than 95% of the papers, from a range of published 
sources, had been un-replicated, un-duplicated, and un-tested, in the average 7.4 
years since their initial appearance. It appears that the complaint expressed in 
the research literature about the rarity of replication and application research is 
valid - an important finding of this work. 

Also potentially problematic was the finding that most of the 
applications/replications studied here proceeded from a positive endorsement of 
the original work, fewer from a neutral view, and none at all from a negative 
one. Again as noted (and deplored) in the literature, this finding suggests that 


21 


replicators identified work consistent with their own objectives or expectations, 
and did not address less compliant issues. It is a question for further research 
whether this finding is general, and whether one of the interpretations is that 
researchers exhibited too slavish an acceptance of, and agreement with, the work 
of others. 

Researchers have addressed the problem of lack of replication with 
suggestions of varying insight and feasibility. Freese;s (2007) suggestion is that 
young academics and graduate students might engage in replication research as 
a way to augment their publications, while testing and presumably becoming 
more familiar with the received wisdom of the field. Doing so could, in fact, be 
enriching, as well as meeting a requirement for more replication research; 

Molina- Azorin (2012) sees replication (using both qualitative and quantitative 
elements, whether produced by new academics or not) as potentially leading to 
more mixed-methods approaches, which in turn could result in stronger 
inferences, and more references than mono-method inquiries typically produce 
(p. 33). The likelihood and feasibility of these processes and outcomes remains to 
be debated and further researched in the field. 

There is continued concern that journal editors may not seek out or 
encourage application/replication research. The literature contains many warnings 
and complaints about this alleged state of affairs. To the degree the charge is 
true it is unfortunate, for the reasons mentioned above. ODL requires 
examination of its roots and assumptions; application/replication research is one 
way to accomplish that objective. Simply said, journal editors and reviewers 
(and funding agents) should be more open to the conduct and reporting of 
application/replication studies, and to contrary, negative, or null findings (which 
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can be a form of rejection). Researchers, for their part, should be - or become - 
better writers and presenters of their work (Holdaway, 1986; Zawicki-Richter et 
al„ 2009). 
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